Augmented reality vehicle interfacing

ABSTRACT

Among other things, an apparatus includes a processor, and storage for instructions executable by the processor to, in connection with a trip of a person in an autonomous vehicle, select a specific location at which the person will be picked up for the trip or a specific location at a destination of the trip, and present through a user interface of a device visible information that depicts the specific location.

CLAIM OF PRIORITY

This application claims priority under 35 USC § 119(e) to U.S.Provisional Patent Application Ser. No. 62/629,764, filed on Feb. 13,2018 and U.S. Provisional Patent Application Ser. No. 62/613,664, filedon Jan. 4, 2018, the entire contents of both which are herebyincorporated by reference.

BACKGROUND

This description relates to augmented reality vehicle interfacing.

The use of augmented reality with vehicles has become a topic ofinterest as suggested by the article “Augmented Reality's Future Isn'tGlasses. It's the Car,”https://venturebeat.com/2017/08/23/ar-will-drive-the-evolution-of-automated-cars/.

SUMMARY

In general, in an aspect, an apparatus includes a processor, and storagefor instructions executable by the processor to (a) receive informationabout a trip of an autonomous vehicle, (b) identify a real-world scenethat corresponds to the trip of the autonomous vehicle, and (c) cause apresentation on a device of augmentation elements for the trip of theautonomous vehicle, the augmentation elements to be presented inlocations that enable the user of the device to view the graphicalaugmentation elements in the vicinity of the visible elements in thereal-world scene.

Implementations may include one or a combination of two or more of thefollowing features. The device includes an augmented reality viewingdevice. The device includes augmented reality glasses. The deviceincludes a head up display. The device includes a display screen of amobile device. The device includes a monitor located remotely from theautonomous vehicle. The user views the real-world scene live and theaugmentation elements are presented on or in the vicinity of thereal-world scene. The real world scene and the augmentation elementsboth are presented on the device. The device is inside the autonomousvehicle. The device is outside of and in the vicinity of the autonomousvehicle. The device is located at a teleoperation facility. Theaugmentation elements include markers that identify moving objects inthe real world scene. The augmentation elements include graphicalrepresentations of moving objects in the real-world scene. Theaugmentation elements include a planned trajectory of the autonomousvehicle. The augmentation elements include an actual specific pick uplocation. The augmentation elements include a marker of a particularautonomous vehicle that is to pick up the user, among two or moreautonomous vehicles visible to the user in the real-world scene.

In general, in an aspect, an apparatus includes a processor, and storagefor instructions executable by the processor to (a) in connection with atrip of a person in an autonomous vehicle, select a specific location atwhich the person will be picked up for the trip or a specific locationat a destination of the trip, and (b) present through a user interfaceof a device visible information that depicts the specific location.

Implementations may include one or a combination of two or more of thefollowing features. The specific location includes a specific physicallocation on or near a road. The specific location includes a specificphysical location identified prior to a time when the person requestedthe trip. The visible information includes a real image of the specificphysical location. The visible information includes a real image of anautonomous vehicle presented with a real image of the specific physicallocation. The specific location is not identified by a street address.

In general, in an aspect, an apparatus includes a mobile deviceincluding a display, a processor, and an app or a browser to cause theprocessor to present on the display a depiction of an actual specificlocation at which a person will be picked up by an autonomous vehiclefor a trip or a specific location at a destination of the trip, theactual specific location having been determined prior to a request bythe person for the trip.

In general, in an aspect, an apparatus includes a processor, and storagefor instructions executable by the processor to (a) receive through auser interface of a device a signal from a user with respect to a tripof the user in an autonomous vehicle, the signal indicating a feature ofthe autonomous vehicle that can be controlled in response to the signalfrom the user, and (b) determine an action to take with respect to theautonomous vehicle to respond to the signal from the user by controllingthe feature of the autonomous vehicle.

Implementations may include one or a combination of two or more of thefollowing features. The feature of the autonomous vehicle that can becontrolled in response to the signal from the user includes atemperature inside the autonomous vehicle. The signal received from theuser includes a temperature inside the autonomous vehicle. The featureof the autonomous vehicle that can be controlled in response to thesignal from the user includes a passenger capacity of the vehicle. Thesignal received from the user includes a number of passengers for thetrip. The feature of the autonomous vehicle that can be controlled inresponse to the signal from the user includes a state of anentertainment system inside the vehicle. The state of the entertainmentsystem includes at least one of a type of entertainment content. Thestate of the entertainment system includes an identifier of a broadcaststation. The signal received from the user includes an identification ofa source or type of content for the entertainment system. The feature ofthe autonomous vehicle that can be controlled in response to the signalfrom the user includes accommodation of a child who requires a childseat. The signal received from the user includes an indication of achild to be present on the trip. The feature of the autonomous vehiclethat can be controlled in response to the signal from the user includesaccommodation of packages for the trip. The signal received from theuser includes of information about packages for the trip. Theinstructions executable by the processor are to select an autonomousvehicle from among two or more available autonomous vehicles based onthe signal from the user. The selection of the autonomous vehicle isbased on at least one of the passenger capacity, the package capacity,or the availability of a child car seat. The processor and storage arepart of a central AV system. The instructions executable by theprocessor are to communicate the action to the autonomous vehicle.

In general, in an aspect, an apparatus includes a mobile deviceincluding a display, a processor, and an app or a browser to cause theprocessor to present on the display at least one option to control afeature of an autonomous vehicle for a trip, the feature including atleast one of a passenger capacity, a package capacity, availability of achild car seat, a temperature inside the vehicle, or a feature of anentertainment system.

In general, in an aspect, an apparatus includes a processor, and storagefor instructions executable by the processor to (a) receive informationabout a real-world environment through which an autonomous vehicle isproceeding, (b) identify one or more moving objects in the real worldenvironment, and (c) cause a user interface of a device to present to aperson a scene that includes a current visible representation of thereal-world environment and visible indications confirming that themoving objects have been identified.

Implementations may include one or a combination of two or more of thefollowing features. The information received about the real-worldenvironment includes an image from an image-capture device and therepresentation presented to the person includes the image. Therepresentation presented to the person includes a schematicrepresentation of the real-world environment. The schematicrepresentation of the real-world environment includes a graphicalrepresentation of the road network. The representation presented to theperson includes a schematic representation of the planned trajectory ofthe vehicle. The visible indications confirming that the moving objectshave been identified include schematic arrays of graphical elementsbased on signals from lidar, radar, or a camera. The visible indicationsconfirming that the moving objects have been identified include markersof the moving objects on a live video presentation of the real-worldenvironment. The markers include graphical boxes. The executableinstructions are to cause the user interface to present the scenesimultaneously with a second scene, the scene including a live video ofthe real-world environment, the second scene including schematicelements corresponding to the real-world environment. The deviceincludes a mobile device. The device includes a work station. The deviceis in the autonomous vehicle. The device is outside the autonomousvehicle. The device is at a teleoperation location. The apparatus of anyof the preceding claims in which instructions executable by theprocessor are to categorize the identified moving objects in thereal-world environment. The markers differ for different categories ofthe identified moving objects.

In general, in an aspect, an apparatus includes a head up displayinterposed between an occupant of an autonomous vehicle and a viewexperienced by the occupant of a real-world environment of theautonomous vehicle, a processor, and storage for instructions executableby the processor to (a) identify objects in the real world environmentthat are in motion, and (b) cause a presentation on the head up displayof indications of objects in the real world environment that are inmotion, the indications being visible to the occupant in proximity toobjects in the real world environment.

Implementations may include one or a combination of two or more of thefollowing features. The indications are graphical elements representingthe moving objects. The executable instructions are to cause thepresentations of the moving objects in the head up display to becoordinated with changes in the position of the autonomous vehicle. Thehead up display includes an effectively transparent screen on which thegraphical elements representing the moving objects are displayed. Thehead up display is interposed between the occupant and the windshield ofthe vehicle. The head up display is interposed between the occupant anda window of the vehicle. The screen includes a coating on a windshieldor window of the vehicle.

In general, in an aspect, an apparatus includes: a vehicle including aplurality of sensors; a processor; and a display configured by theprocessor to allow a user to view information with respect to thevehicle; wherein the processor receives data from the plurality ofsensors about an environment of the vehicle, analyzes the data togenerate the information with respect to the vehicle, and shows theinformation on the display, and wherein the information includesfeatures relevant to operation of the vehicle that not present in theenvironment of the vehicle.

In general, in an aspect, a vehicle includes: driving componentsincluding an acceleration component, a steering component, and adeceleration component; autonomous driving capabilities to issue signalsto the driving components to drive the vehicle in an at least partialautonomous driving mode; an analysis component to analyze dataaccumulated by sensors on the vehicle and generate information about anenvironment of the vehicle, and a display that shows the informationabout the environment of the vehicle, wherein the information includesfeatures not present in the environment of the vehicle.

In general, in an aspect, a method includes: receiving data from one ormore sensors about the surroundings of the one or more sensors;analyzing the data to identify one or more features in the surroundingsof the one or more sensors, the one or more features related tooperation of an autonomous vehicle (AV); and displaying, on a screen, arendering of the surroundings of the one or more sensors, wherein therendering includes data from at least one of the sensors, and at leastone object representing one or more of the identified features.

Implementations may include one or a combination of two or more of thefollowing features. The rendering includes at least one imaginary objectrelated to operation of the autonomous vehicle. The imaginary objectincludes a rendering of a vehicle. The imaginary object includes arendering of a vehicle located at a location chosen by a user. Theimaginary object includes a marking indicating a planned trajectory ofthe autonomous vehicle. The one or more sensors are attached to aheadset. The one or more sensors are attached to a pair of glasses. Theone or more sensors are attached to a smartphone. The at least oneobject is a tag that identifies a class to which the identified featurebelongs.

Implementations may include one or a combination of two or more of thefollowing features. Displaying a rendering includes presenting aninteractive interface. Presenting an interactive interface includespresenting a field of view or a bird's-eye of a vision sensor of thevehicle. Presenting an interactive interface includes presenting currentperception information. Presenting an interactive interface includespresenting current or past or both motion planning information.Presenting an interactive interface includes presenting a system diagramof the vehicle, the system diagram including one or more hardwarecomponents, or one or more software processes, or both. The dataincludes one or more of the following: a map, sensor data in the vehicleor a related AV system, trajectory data in the vehicle or a related AVsystem, or vision data in the vehicle or a related AV system. Thedisplay is within the autonomous vehicle. The display is remote to theautonomous vehicle.

These and other aspects, features, and implementations will becomeapparent from the following descriptions, including the claims and canbe expressed as methods, apparatus, systems, components, programproducts, methods of doing business, means or steps for performing afunction, and in other ways.

DESCRIPTION OF THE DRAWINGS

FIGS. 1, 6A, 5A, 7, 8 and 9 are block diagrams.

FIGS. 2A-2C, 3A-3J, 4B-4F, and 6B-6C are screen shots

FIGS. 4A, 5B-5E are schematic diagrams.

DETAILED DESCRIPTION Overview

As shown in FIG. 8, the operation, hailing, occupancy, driving, andother activities of people 10 in using, viewing, and avoiding accidentswith one or more vehicles 12 (such as but not limited to an autonomousvehicle) on a system 200 can be associated with augmented realitytechniques 13 implemented on or by a variety of user interface devices14 such as a mobile device 16, a head-worn interface device 18, a headup display, or an in-vehicle display 20, among others. The userinterface devices may be inside or outside the vehicle. In someimplementations, the augmented reality techniques include presentationof augmented reality elements 22 that are superimposed on a real-worldscene 24. In some cases, the augmented reality elements 22 aresuperimposed on the real-world scene 24 by presentation through userinterfaces 26 on displays 27 of the user interface devices. In someinstances, the augmented reality elements are superimposed on thereal-world scene by presentation through the user interfaces of both theaugmented reality elements 22 and a view 28 of the real-world scene,which may have been captured by a camera, microphone, or other scenecapture device 30, or combinations of them. The user interface devicescan present the augmented reality elements (and in some cases thereal-world scene) using native applications (or Web browsers) running onthe user interface devices. The native applications (or Web browsers)can be capable of accepting as inputs data representing the real-worldscene, generating augmented reality elements, and combining them forpresentation on the displays of the user interfaces. Often, thereal-world scene and the augmented reality elements are presented inreal time (e.g., “live”) so that they are relevant and useful to peoplein real-world contexts associated with vehicles.

The term “augmented reality” or “AR” is used broadly to include, forexample, any direct or indirect view of a real-world scene that issupplemented, enhanced, amplified, extended, or otherwise “augmented” bypresentation of any sensory element that is not physically part of thereal-world scene. The sensory element can be visual such as video,graphics, or GPS data, haptic, or auditory, or a combination of two ormore of those, among other kinds of elements.

Autonomous Vehicle Example

We will frequently use the example of an autonomous vehicle as thecontext for our descriptions. Nevertheless, at least some of thetechnologies that we describe here may be applicable to and useful withvehicles driven by people.

The term “autonomous vehicle” or “AV” or “driverless car” or“self-driving car” is used broadly to include, for example, any vehiclethat has one or more autonomous driving capabilities.

The term “autonomous driving capability” is used broadly to include, forexample, any function, feature, or facility that can participate in thedriving of an AV other than by a person manipulating a steering wheel,accelerator, brake, or other physical controller of the AV.

The technologies that we describe here may be applicable to any vehiclethat has one or more autonomous driving capabilities including fullyautonomous vehicles, highly autonomous vehicles, and conditionallyautonomous vehicles, such as so-called Level 5, Level 4 and Level 3vehicles, respectively (see SAE International's standard J3016: Taxonomyand Definitions for Terms Related to On-Road Motor Vehicle AutomatedDriving Systems, which is incorporated by reference in its entirety, formore details on the classification of levels of autonomy in vehicles).Autonomous driving capabilities may attempt to control the steering orspeed of the vehicles. The technologies described in this document alsocan be applied to partially autonomous vehicles and driver assistedvehicles, such as so-called Level 2 and Level 1 vehicles (see SAEInternational's standard J3016: Taxonomy and Definitions for TermsRelated to On-Road Motor Vehicle Automated Driving Systems). One or moreof the Level 1, 2, 3, 4 and 5 vehicle systems may automate certainvehicle operations (e.g., steering, braking, and using maps) undercertain driving conditions based on processing of sensor inputs. Thetechnologies described in this document can benefit vehicles in any ofthe levels, ranging from fully autonomous vehicles to human-operatedvehicles.

An autonomous vehicle typically is capable of sensing its environmentand navigating through that environment without human input or withreduced human input as compared to a person navigating a traditionalvehicle. Autonomous vehicles use a variety of techniques to detect theirsurroundings, such as radar, laser light, GPS, odometry, and computervision, among others, and to produce corresponding sensory information.Advanced control systems interpret this sensory information to identifyappropriate navigation paths, as well as obstacles and relevant signage.Autonomous vehicles can reduce mobility and infrastructure costs, andincrease safety resulting in a significant reduction in trafficcollisions and their resulting injuries. Autonomous vehicles mayfacilitate a variety of business models for transportation services,especially via the sharing economy.

As shown in FIG. 1, a typical activity of an autonomous vehicle (AV) 100is to safely and reliably drive autonomously or partially manually orboth along a trajectory 198 through an environment 190 toward adestination 199 while avoiding objects (e.g., mountains 191, vehicles193, pedestrians 192, cyclists, and other obstacles) and obeying rulesof the road (e.g., rules of operation or driving preferences). Thefeatures, functions, and facilities of an AV 100 or an AV system 120that enable the AV 100 to perform the autonomous driving often arereferred to as autonomous driving capabilities.

The term “trajectory” is used broadly to include, for example, any pathor route from one place to another; for instance, a path from a pickuplocation to a drop off location; a path toward a goal position.

The term “goal” or “goal position” is used broadly to include, forexample, any place to be reached by an AV, including, for example, aninterim drop-off location, a final drop-off location, or a destination,among others.

The driving of an AV 100 typically is supported by an array oftechnologies (e.g., hardware, software, and both stored and real-timedata) that in this document together with the AV system 100 is referredto as an AV system 120. In some implementations, one or some or all ofthe technologies are onboard the AV 100. In some cases, one or some orall of the technologies are at another location such as at a server(e.g., in a cloud computing infrastructure).

Components of an AV system 120 can include one or more or all of thefollowing:

1. Functional devices 101 of the AV 100 that are instrumented to receiveand act on commands for driving (e.g., steering 102, acceleration,deceleration, gear selection, and braking 103) and for auxiliaryfunctions (e.g., turning signal activation) from one or more computingprocessors 146 and 148.

2. Data storage unit 142 or memory 144 or both for storing machineinstructions or various types of data or both.

3. One or more sensors 121 for measuring or inferring, or both,properties of the AV's state or condition, such as the AV's position,linear and angular velocity and acceleration, and heading (e.g., anorientation of the leading end of the AV). For example, such sensors caninclude, but are not limited to: GPS; inertial measurement units thatmeasure both vehicle linear accelerations and angular rates; individualwheel speed sensors for measuring or estimating individual wheel slipratios; individual wheel brake pressure or braking torque sensors;engine torque or individual wheel torque sensors; and steering wheelangle and angular rate sensors.

4. One or more sensors for sensing or measuring properties of the AV'senvironment. For example, such sensors can include, but are not limitedto: monocular or stereo video cameras 122 in the visible light, infraredor thermal (or both) spectra; lidar 123; radar; ultrasonic sensors;time-of-flight (TOF) depth sensors; speed sensors; and temperature andrain sensors.

5. One or more communications devices 140 for communicating measured orinferred or both properties of other vehicles' states and conditions,such as positions, linear and angular velocities, linear and angularaccelerations, and linear and angular headings. These devices includeVehicle-to-Vehicle (V2V) and Vehicle-to-Infrastructure (V2I)communication devices and devices for wireless communications overpoint-to-point or ad hoc networks or both. The communications devices140 can communicate across the electromagnetic spectrum (including radioand optical communications) or other media (e.g., air and acousticmedia).

6. One or more communication interfaces 140 (e.g., wired, wireless,WiMAX, Wi-Fi, Bluetooth, satellite, cellular, optical, near field, orradio, or combinations of them) for transmitting data from a remotelylocated database 134 to the AV system 120, transmitting sensor data ordata related to driving performance to a remotely located database 134,or transmitting information that relates to teleoperations, or acombination of them.

7. One or more data sources for providing historical, or real-time, orpredictive information, or a combination of any two or more of themabout the environment 190, including, for example, maps, drivingperformance, traffic congestion updates or weather conditions. Such datamay be stored on a data storage unit 142 or memory 144 on the AV 100, ormay be transmitted to the AV 100 via a communications channel from aremote database 134 or a combination of them.

8. One or more data sources 136 for providing digital road map data fromGIS databases, potentially including one or more of the following:high-precision maps of the roadway geometric properties; maps describingroad network connectivity properties; maps describing roadway physicalproperties (such as traffic speed, traffic volume, the number ofvehicular and cyclist traffic lanes, lane width, lane trafficdirections, or lane marker types and locations, or combinations ofthem); and maps describing the spatial locations of road features suchas crosswalks, traffic signs or other travel signals of various. Suchdata may be stored on a memory 144 on the AV 100, or transmitted to theAV 100 via a communications channel from a remotely located databaseserver, or a combination of the two.

9. One or more data sources 136 or sensors 132 for providing historicalinformation about driving properties (e.g., speed and accelerationprofiles) of vehicles that have previously traveled along local roadsections, for example, at similar times of day. Such data may be storedon a memory 144 on the AV 100, or transmitted to the AV 100 via acommunications channel from a remotely located database 134, or acombination of the two.

10. One or more computing devices 146 and 148 located on the AV 100 (orremotely or both) for executing algorithms for on-line generation ofcontrol actions based on both real-time sensor data and priorinformation, allowing the AV system 120 to execute its autonomousdriving capabilities.

11. One or more processes of processing sensor data, perceiving theenvironment, understanding conditions that are currently presented byand may at future times be presented by the perceived environment,performing trajectory planning, performing motion control, and makingdecisions based on those perceptions and understandings. A process maybe implemented by integrated circuits, field-programmable gate arrays,hardware, software, or firmware, or a combination of two or more ofthem.

12. One or more interface devices 150 (e.g., displays, mice, trackballs, keyboards, touchscreens, speakers, biometric readers, and gesturereaders) coupled to the computing devices 146 and 148 for providinginformation and alerts to, and receiving input from, a user (e.g., anoccupant or a remote user) of the AV 100. The coupling may be wirelessor wired. Any two or more of the interface devices may be integratedinto a single device.

Other features and components can also be integrated into the AV system120.

Augmented Reality in Vehicle Interfacing

The technologies that we describe here have a wide variety ofapplications in vehicle interfacing. The applications can be inside avehicle or outside the vehicle and can be useful to drivers, passengers,people hailing vehicles, pedestrians, and other people. Below wedescribe a few applications as examples.

Requesting Transportation Services

In some applications, such as ride sharing services, autonomous andother vehicles can provide transportation services to users who do notown the vehicles. In such examples, techniques are provided to enableusers to request or a hail the vehicles. In some cases, thetransportation services are requested or the vehicles are hailed using,for example, user interfaces presented on mobile devices on which nativeapplications or Web browsers are running. Some aspects of such userinterfaces can include augmented reality features.

For example, referring to FIGS. 2A-2C, a user 202 can request a vehicletrip in an AV 100 that operates as part of an AV transportation servicessystem 200 (FIG. 8). The user requests the vehicle trip using a display203 on the mobile device or other human-computer interface device 150that the user 202 uses to communicate with the AV system 120. Forexample, the user interface 201 can be part of a mobile phoneapplication (or Web pages) that is displayed on a smartphone acting asthe human-computer interface device 150, and operates as part of an AVhailing system 200.

The user 202 can interact with the AV transportation services system ina variety of ways; for example, a request for a ride can be made basedon a user's gesture, voice, typing, a controller, a 2D/3D userinterface, such as a 2D/3D augmented reality. In the example shown inthe figures, the user 202 uses a series of user interface screensbeginning with a start screen 201 on a smartphone app. Such a screen hasan upper portion 204 that can include a menu access icon 205, statusupdates 207, and other header information. A typical screen, includingstart screen 201, also has a lower portion 206, which displays suchinformation as a map 208 of the user's 202 location and augmentedreality elements such as enhancement features 220 (not shown in FIGS. 2Athrough 2C) discussed in more detail below.

Landing Page

Referring to FIG. 2A, the illustrated screen display is an initiallanding page or start screen 201 of an app or website, where the user202 is shown a map 208 in the vicinity of his or her current locationand an icon 209 showing his or her current location with high accuracy(e.g., within 1 m, within 10 cm), and optionally other relevantinformation including the neighboring road network and geophysicalfeatures, and icons 213, 215, 217 showing the locations of nearby AVs100 that can be hailed. The upper portion 204 of the screen alsoincludes information such as a prompt box 210 that in the case of startscreen 201 invites the user 202 to select a destination. The user 202can type or otherwise input a destination, or a suggestion screen 211(FIG. 2B) can display icons to 19 that suggest nearby destinations whilethe prompt box 210 shows a request that the user choose one of thedisplayed potential destinations. The suggestion screen 211 can alsoinclude additional information 207 in the lower portion 206 of thescreen, in this case information about one of the illustrated selectabledestinations. In FIG. 2C, a destination 199 has been selected, itsaddress 222 is shown, and a pickup location of one of the availablevehicles is highlighted 214. FIG. 2C also includes a button 223 that theuser can invoke to set a pickup location and a view of a currentlyhighlighted pickup location showing its address and other information.Computing processors (e.g., computing processors 146 and 148) canexecute instructions to select and show address 222, includinginstructions stored in memory 144.

Pickup Location and Trip Details Pages

Referring to FIGS. 3A-3J, the user 202 can specify further detailsconcerning his or her pickup location 214 and details about the upcomingtrip to the destination 199. A pickup screen 231 allows the user 202 tochoose his or her pickup location using a second prompt box 212. Theuser's 202 current location 237 is shown on the map 208 on pickup screen231, which is localized to the more immediate vicinity of the user athis or her current location than on start screen 201. The map 208displays possible alternate pickup locations 214, 216 (labeled bynumbers 1 and 2) where the AV 100 can stop near the user's currentlocation 237 for him or her to enter the stopped AV 100 and proceed tothe destination 199.

The two possible alternate pickup locations 214, 216 displayed on themap 208 are chosen by the AV transportation services system 200 usingand combining information from various sources such data sources 136 andsensors 121. The AV transportation services system 200 analyzes theinformation, in conjunction with the user's 202 current location (his orher general pickup location 222), to determine potential pickuplocations 214, 216 that conform to safety rules and road restrictionsand user preferences, among other things. As shown, the potential pickuplocations 214, 216 are specific coordinates at which the AV 100 willstop so that the user 202 may enter. In some instances, the AVtransportation services system 200 may include predetermined pickuppoints in its data storage unit 142 or memory 144 that are within anacceptable range or walk distance to the user's location (e.g., within 2minutes of walking, or within 250 m). These predetermined pickup pointsmay be parking spots or curb spots that are known to be safe locationsat which the AV 100 may stop while waiting for the user to enter.

In pickup screen 231 two potential pickup locations 214, 216 are shown,although just one, or more than two potential pickup locations are alsopossible. In pickup screen 231, a first potential pickup location 214 ishighlighted for the vehicle's approach from left of the user's currentlocation and a second potential pickup location 216 is highlighted forif the vehicle were to approach from right to the user's currentlocation. Two locations are illustrated, but one location, or more thantwo locations are also possible.

The pickup screen 231 has additional information 207 in the lowerportion 206 of screen. The additional information 207 displays elements(for example, augmented reality elements) that are not actually presentin the real world in the vicinity of the user's current location, e.g.,augmented reality (AR). In this example, the additional information 207shows a camera view 218 of the user's general current location. In someinstances, the view is taken from the camera of the smartphone held bythe user, or alternatively the view can be a saved standard view (e.g.,a street view from a service such as Google maps). In the camera view218, the first potential pickup location 214 is displayed with anenhancement feature 220, e.g., an object that does not exist in theuser's real world environment but which is overlaid on the camera view218 so as to appear to belong in the real world as seen in the cameraview. In this instance, the enhancement feature 220 is an image of an AV100 positioned in the first potential pickup location 214. The user isable to evaluate the relationship of the augmented reality element (theAV) relative to the potential pickup location to determine if thepotential pickup location 214 displayed is desired (e.g., near anoverhang if it is raining, or free from obstacles or obstructionstemporarily in the potential pickup location 214). The user 202 selectsthe desired pickup location, here the first potential pickup location214.

The AV hailing system 200 may comprise multiple processes, realized byintegrated circuits, field-programmable gate arrays, hardware, software,or firmware, or a combination of two or more of them. Processing andanalyzing the signals and data may be realized by a processor, or acomputing resource of the processor. The processor can communicate withthe user through an interface and display to display screens such aspickup screen 231. The computing device can be implemented in a numberof different forms, for example, it can be implemented as part ofsmartphone 682.

Referring to FIG. 3B, the user 202 can continue to specify detailsconcerning his or her pickup location and details about the trip to thedestination 199 in trip screen 241. The user 202 has selected a generalpickup location 222, e.g., an address or a cross street, or a point on amap as was shown in FIG. 2B for selection of the destination. Thegeneral pickup location 222 appears in the second prompt box 212. Thetrip screen 241 includes a user details prompt 224 in which the user 202tells the AV transportation services system 200 details about the user,e.g., the number of adult travelers, if any children will be on thetrip, and if luggage will be on the trip. The information given to theAV transportation services system 200 by the user details prompt 224allows the AV transportation services system 200 to choose anappropriate vehicle, e.g., a nearby vehicle that has the spacerequirements matching those input by the user at the user details prompt224.

The upcoming trip display screen 251 in FIG. 3C shows the user 202 theplanned route from the general pickup location 222 to the destination199 on the map 208. Additional information 207 includes an estimateddistance and time for the trip. If the user agrees with the plannedroute, he or she presses the “request ride” button 227. The AVtransportation services system 200 then matches the request (e.g.,identifies which if any vehicles are in the user's vicinity, if theymatch the criteria included in the user details prompt 224 and therequested type of vehicle from the vehicle details prompt 226).Referring to FIG. 3D, at vehicle selection screen 261, the user 202 canselect the type of vehicle desired for the upcoming trip in vehicledetails prompt 226. In some instances, the user 202 will either not beshown or be unable to select certain types of vehicles, for example ifcertain types of vehicles are not available, or if a certain type ofvehicle does not match the information entered at the user detailsprompt 224 by not having the requisite room for travelers or luggage, ornot having a car seat available for child travel

Referring as well to FIG. 9, an AV hailing system 200 may include an AVmatcher 700 (e.g., an optimization algorithm) to determine which AV (ifany) matches the request of the user included in the user details prompt224 and vehicle details prompt 226 with AV fleet information 720 (e.g.,whether each vehicle is in use, current trajectories, data from sensors,or combinations of them) to determine which AV in the fleet of AVs issuitable for the user 202. The AV transportation services system 200identifies the most suitable specific vehicle and displaysvehicle-specific information 228 to the user (FIG. 3E) on selectedvehicle screen 271. The vehicle-specific information 228 can include apicture of the specific vehicle, the license plate of the specificvehicle, and the name of the specific vehicle. The vehicle-specificinformation 228 may be visual or aural, or both. The vehicle-specificinformation 228 also may be transmitted to a remote recipient orcomputing device 730, such as a remote operator, or another autonomoussystem.

As shown in FIGS. 3F and 3G of screens 281 and 291, the map 208 can showdetails of the AV vehicle's 100 approach, such as its current position232, and the current trajectory 198 on the way to the selected pickuplocation 214. The screens can show arrival information 234, such as thetime remaining until the vehicle arrives at the pickup location 214.

In FIG. 3G, the user 202 can continue to specify details concerning thetrip to the destination 199 at vehicle customization screen 281. Avehicle-specific prompt 230 allows the user to customize his or herride, including selecting music (e.g., a radio station or music serviceif one is available in the selected vehicle) or the temperature of thevehicle. The vehicle-specific prompt 230 will show differing selectionsand information for different given vehicles and their features (e.g.,control of seat warmer(s), windows being up or down, position of theseats forward or backwards, etc.).

In FIG. 3H, the AV vehicle 100 has arrived and is positioned at thepickup location 214. An enhancement feature 220 shows the car in thechosen pickup spot 214, which now matches the actual view as seen by theuser, thus arrival screen 311 has an enhancement feature 220 that shouldcorrespond with reality. The screens 281, 291 can also instruct therider as to how to reach the pickup location 214, and the arrival screen311 tells the user when the AV vehicle 100 has arrived (FIG. 3H).

Referring to FIG. 3I, a trip screen 321 is displayed on interface device150 while the to user 202 is inside the vehicle and on the way to thedestination 199. The trip screen 321 shows the current position 232 ofthe AV 100 on the map 208. During the trip, the user 202 can interactwith the human-computer interface on the interface device 150 that theuser has been using, e.g., a smartphone.

In FIG. 3J, the user 202 has arrived at his or her destination, and theinterface device 150 requests feedback about the trip. The AVtransportation services system 200 may store the user's information(e.g., in memory 144) to recommend future AV booking or riding. The AVtransportation services system 200 also records the rides, for example,to more easily keep track of any items that are inadvertently left in avehicle after a user departs.

In-Vehicle Augmented Reality Experiences

FIG. 4A shows a user 202 seated in the backseat of an AV interior 104.An in-vehicle display 331 can be located (e.g., mounted) on anin-vehicle display device 341 inside the AV 100 and within view of theuser when seated inside (for example, near the back of the passengerseat facing a backseat rider as shown). The in-vehicle display 331 canbe controlled using a variety of technologies, for example, gesture,voice, typing, controller, a 2D/3D user interface such as a 2D/3Daugmented reality interface, or combinations of them, and can bedisplayed, for example, on a tablet, or projected onto a window orwindshield 364 of the AV 100. If no display in projected onto thewindshield 364, then the normal real world of the environment 190 of theAV 100 is shown through the windshield 364, as in the figure.

Information about the trip in progress can be shown on the interfacedevice 150 or the in-vehicle display 331, or both, including the resultsof a safety check (e.g., if seat belts are fastened, if a child listedfor the trip is secured in the child seat, if the doors are locked). Insome instances, the user 202 can use the camera of their smart phone fora visual check on safety checks, e.g., the user can take a picture of achild sitting in a legally-required safety seat and upload the image tothe AV transportation services system 200. In some instances, varioussensors 121 located within the vehicle or cameras 122 within the vehiclecan record the safety check (e.g., seatbelt sensors or in-vehicle camerathat automatically takes and uploads a picture of a child securelyfastened in a legally required safety seat).

Referring to FIGS. 4B and 4C, the in-vehicle display device 341 (and/orthe interface device 150) of FIG. 4A can display a three-dimensionalperspective view of the vicinity 343 of the vehicle 345 in the directionof travel, including the road 347 currently being traveled, intersectingroads 349, a perspective view of the vehicle at its current position 232in the context of direction of travel, objects 360 in the drivingenvironment, and an approaching portion of the trajectory 198 toward thedestination 199. This view can be a schematic view 358 or a picturedisplay 368 or both. The objects 360 depicted in the view can be objectsthat have been detected in real time by the sensors 121 that are part ofthe AV system 120, including cameras 122.

FIGS. 4B and 4C show two examples of the in-vehicle display 341; thein-vehicle display 341 is continually updated in real time to showinformation about objects identified in the current sensor data as theAV 100 moves and as objects (such as pedestrians or other vehicles)around the AV 100 move.

This depiction in both the schematic view 358 and the picture display368 provides comfort to users who may be unsure about riding in avehicle without human operation. The depiction also can be informativeand instructive about the operation of the vehicle. The cameras 122detect 360 degrees around the vehicle and therefore the views canpresent information within a 360° range. Also depicted is tripinformation 370, including the current position, the destination, thetime and distance remaining on a trip, and the current time, among otherthings.

In the schematic view 358, the objects 360 are illustrated using dots,dashes, or other abstract representations of information received bylidar or radar sensors. As the vehicle progresses along the trajectory,these arrays of dots, dashes, or other abstract representations changecontinuously to reflect the changing data received from the lidar orradar sensors.

The picture display 368 of the in-vehicle display 341 includes areal-time video camera presentation 369 of the real-world scene in thedirection of travel of the vehicle. The picture display 368 includesenhancement features 220. In the example figures, the enhancementfeatures 220 are red boxes or tags 362 identifying features or objects360 in the vehicle surroundings. The enhancement features 220 areoverlaid on real data, e.g., tags 362 are overlaid on visual data fromone of the cameras 122 mounted to the AV 100. The in-vehicle display 341thus shows augmented reality, that is, a camera view supplemented byenhancement features identifying one or more objects shown in the cameraview. In some instances, the appearance of the enhancement features 220can change depending on the types of objects identified by the AV system120. For example, the tags 362 can have a color, or shape, or outline,or other identifying marker that differentiates different classes ofobjects such as pedestrians, vehicles, or traffic control features suchas cones or traffic lights.

A key function of the tags 362 is to demonstrate to the viewer whichobjects in the environment, in particular moving objects, have beenidentified by the AV system as the vehicle proceeds. Demonstrating thisidentification of objects including moving objects helps to comfortriders by implying that the sensors of the AV system are capable ofidentifying objects that may be hazards for the vehicle.

In addition to this depiction of objects detected by sensors 121providing comfort to passengers riding in the vehicle, the enhancementfeatures 220 can be used by a vehicle operator (either in the vehicle,or remotely), and by system operators such as safety engineers. A systemoperator viewing the in-vehicle display 341 can analyze and assess theefficacy of the AV system's ability to detect and identify objectsdetected by the sensors 121. A system operator can also view anin-vehicle view of a camera inside the AV 100.

In general, operating an autonomous system includes planning itsmotions. A trajectory may be associated with multiple motion actions(e.g., accelerate, maintain a speed, decelerate, change orientation,stop, follow traffic signals, and avoid hitting objects) that will beexecuted by an autonomous system in a driving environment to achieve thetrajectory. Some motion actions may be performed in parallel (e.g.,changing an orientation and deceleration), and some motion actions maybe performed in series (e.g., acceleration and then deceleration). Forinstance, operating an autonomous system 120 along the trajectory 198may include the following: accelerate at start, slow down and make aright turn at a given location, maintain a slow speed, make a left turnat a second location when the traffic signal allows the left turn,accelerate, decelerate at, and stop at a goal position. Implementationsof the motion planner may include a trajectory identifier. Thetrajectory identifier may analyze a map for an autonomous system tonavigate from a start position and a goal position. The map can shownon-drivable regions, and other vehicles on the road. To identifypossible trajectories, the trajectory identifier may begin by samplingthe map. The samples that are in the non-drivable regions or blocked byan object (e.g., a vehicle) are then removed. Based on the remainingsampling points, the trajectory may identify multiple candidatetrajectories. Safety engineers developing such a motion planningalgorithm could be assisted by AR features (enhancement features 220)included in the map. Such a safety engineer may be present in the AV 100and accessing a system diagram of the vehicle or may be remote from theAV 100.

Among other things, a risk monitoring process may identify risks bymonitoring an environment near the AV, an operation of the AV system, orthe interior of the AV, or combinations of them. For instance, analyzingsignals from sensors 121 (e.g., a vision sensor, a lidar or a radar, orcombinations of them) can produce information about other objects (e.g.,vehicles, infrastructure, and pedestrians) in the environment; examplesof such information include: locations, speeds, orientations,boundaries, sizes, dimensions, traffic lights, manufacturers, platenumbers, owners, drivers, and vehicle operations. The information may beanalyzed to e.g., predict a potential collision or detect an existingcollision. Analyzing a map from a database or images from a visionsensor, or both, can further determine foreground and background. Forexample, a map used by the AV system 120 may encode information about anelevation profile of the road surface. This information can be used toclassify a given point as belonging to the road surface by analyzingdepth information, acquired from a vision sensor (e.g., a stereo camera)and applying segmentation to identify a background region, or aforeground object, or both. Safety engineers developing a riskmanagement algorithm could be assisted by AR features (enhancementfeatures 220) included in the map. For example, the system may includeenhancement features 220 on a real world view of the road surface imagethat distinguish points as belonging to the background or foreground. Asafety engineer viewing such an image can easily classify the points andreclassify the points if they have been misidentified, improving theaccuracy of the detection algorithm.

Referring to FIGS. 4D and 4E, an AR screen 362 can be part of thein-vehicle display that is visible on a windshield 364 of the AV 100. Inthis instance, the windshield 364 is largely a conventionallysee-through surface, and actual vehicles 193 are visible through themost of the windshield 364. In addition, the AR screen 262 occupies abottom portion of the windshield 364, for example a projector 366 withinthe AV 100 projects information and enhancement features 220 into the ARscreen 262. In some instances, the AR screen 262 is at least partly seethrough. Information can be displayed on the AR screen 262, such as tripinformation 370. The AR screen 262 can also display enhancement features220; in this instance the enhancement features 220 are markers or tagsindicating that a vehicle 193 (or other object) has been detected by thesensors 121 of the AV system 120 at a position near or above theposition of the marker, and identified or classified as a vehicle. Theenhancement feature 220 can be different for each type or class ofobject identified. For example, the marker or tag may have a differentcolor or different shape to identify a pedestrian, another vehicle, atraffic cone, etc. In some instances, instead of a separate physicalscreen (or in addition to it), the bottom portion of the windshield 364can be coated with a coating that allows information to be displayed onit; for example, the coating may allow the information for the AR screen262 to be projected onto the glass of the windshield 364.

The user interface information presented on the in-vehicle displayscreen 331 or on the AR screen 262 can be shared with other devices. Forexample, the information can be shared to a screen on a smartphone, forexample, a smartphone of a waiting family member, or can be shared to ascreen being observed by a teleoperator or a fleet manager as discussedbelow.

In some instances, the in-vehicle display 341 can display a virtualscene such as a nature scene. Instead of the enhancement features 220shown in FIG. 4D that indicate the position and operation of the AVsystem 100, such a virtual scene can allow a passenger within the AV 100to experience a virtual environment (as his or her attention is notnecessary for operation of the vehicle). The virtual scene can bedisplayed on the windows and/or windshield, so the rider may immersethemselves in a virtual environment (e.g., virtual reality inside an AV100). The vehicle could thus appear to be passing through a different (acalming, or exotic) scene instead of the actual environment beingtraversed by the vehicle (e.g., a cityscape).

In some instances, a virtual shield 381 is positioned between seats orbetween in-vehicle displays 331 in the AV interior 104. The virtualshield 381 ensures that the screen of one in-vehicle display 331 (whichcan be an AR headset 380) cannot be seen by another user 202. If thereare multiple users (e.g., carpoolers) in a vehicle, a first passengerwearing AR glasses may have his or her own immersion environment that isseparate and private from the AR environment of a second fellowpassenger. The virtual shield 381 may be advantageous in instances wheremultiple users 202 are sharing the same AV 100. In some instances thevirtual shield 381 can be a privacy screen such as screens that do notallow a viewer to view a screen display greater than a pre-determinedangle relative to the screen. In other instances the virtual shield 381can be a field damper.

The various screen displays discussed above should be considered by wayof example, other information, details, and inputs are also possible andwithin the scope of this disclosure.

FIG. 5A shows a diagram of an augmented reality (AR) headset 380 thatcan operate as part of the AV transportation services system 200. An ARheadset can include a processor 302. The processor receives data fromand returns data to the AV system 120, and receives data from a trackingunit 310. The data processed by the processor 302 is displayed on thescreen present in the AR 380 headset, which is a pair of lenses 304 isused for visualization of the rendered image. The tracking unit tracksthe movement of the user's head via multiple sensors that determineinformation such as the coordinates of the user's orientation, and caninclude a 3 axis gyroscope, a 3 axis accelerometer, a 3 axismagnetometer (or compass), and sensors that detect ambient light sensingand proximity of objects to the sensor. These data can include angularposition (yaw/pitch/roll), velocity and acceleration, as well as linear(x/y/z) position, velocity, and acceleration data. These data are fed tothe processor 302 which generates the real time rendered images. Therendered images are then displayed on the screen or lenses 304, therebycreating stereo visual sensations. The AR headset 380 can also includeone or more cameras 120 capable of video recording, and wirelesstransmitters and receivers (e.g., Wi-Fi and/or Bluetooth). Such an ARheadset 380 detects the real world environment of the user 202,including the orientation of the user and a rendering of what the useris seeing in his or her environment.

FIG. 5B shows a top perspective view and FIG. 5C-E shows 3D schematicviews of a user 202 wearing an augmented reality (AR) headset 380 thatcan operate as part of the AV transportation services system 200. The ARheadset 380 can be goggles or glasses worn by the user 202), which showsenhancement features 220. Such an AR headset 380 can include an opticalhead-mounted display designed in the shape of a pair of eyeglasses.

In the examples shown in the figures, the enhancement features 220 arean overlay of two or more potential pickup locations including potentialpickup locations 214, 216 and the proposed route or trajectory 198 of ahailed AV 100 that is enroute to the user 202. The AR headset 380 addsthese features into the display (e.g., causes them to appear on theglasses of the AR headset 380) to make them appear as if those objectsare in the real world environment. Thus, the user 202 looking throughthe AR headset 380 sees real world features such as buildings orpedestrians 192 that are within a field of view 385 of the headset 380and also sees enhancement features 220 that are displayed on the ARheadset 380 as if the enhancement features 220 are part of the realworld environment.

The AR headset 380 can adapt the enhancement features 220 displayed, andthe view of the enhanced features displayed, as the user changes his orher field of view 385 to include different sightlines and objects. Thegyroscope, accelerometer, and magnetometer within the AR headset 380detect the new orientation and the new field of view that is visible tothe camera within the AR headset 380. Different objects (e.g., differentpedestrians 192) are visible through the headset 380. At the same time,different enhancement features 220 are also visible through the headset380. A processor associated with the AR headset 380 generates thedifferent views of the enhanced features and embeds the AR features intothe real world view in real time. In this example, the enhancementfeature 220 seen through the headset 380 is the portion of thetrajectory 198 that is to be traversed by the hailed AV 100. As the user202 changes his or her orientation, for example to the left, differentportions of the real world environment and enhancement features 220become visible. The system is able detect the actual view of the user'scurrent environment and project accurate enhancement features 20 ontothe glasses for simultaneous viewing appropriate to a given field ofview 385 at a given time.

As seen in FIG. 5E, the enhancement features 220 are made available tothe headset 380. The camera, gyroscope, accelerometer, and magnetometerof the AR headset 380 detect the real world view and orientation of theheadset 380. The user 202 looks through the glasses and sees the realworld field of view, and also appears to see the portion of thetrajectory 198 that is present within the current field of view 385.This enhancement feature 220 is displayed on the glasses of the ARheadset 380 itself so as to overlay the real world features within thefield of view 385 of the user 202 at the given moment.

In some examples, the enhancement features 220 allow a user todistinguish the assigned vehicle from any other vehicles that might bein the same area. In FIG. 5E, two enhancement features 220 are visible,the trajectory 198 and AV marker 387. The AV marker 387 is a specialfeature displayed on the AR headset 380 (e.g., color, flashing lights,spotlight, arrow, etc.) that identifies a specific AV 100. The AV marker387 is of particular use if there are multiple AVs 100 present in aregion; as the user 202 looks through the AR headset 380 the AV marker387 identifies the specific vehicle that has been assigned to that user202.

Teleoperation

On some occasions, such as when an AV is driving on a road andexperiences an event—such as a system fault, extreme weather conditions,or temporary detour—it may be useful to have a remotely located personprovide assistance in the operation of the AV. A teleoperation system,which may be remote or local or a combination of remote and local to theAV or AV system, can enable a teleoperator to interact with the AVsystem (e.g., providing commands, visualizing a driving condition, andinvestigating functionality of a hardware component or software process)through a communication channel. The interactions may assist the AVsystem to adequately respond to various events. The use of teleoperatorsystem with an AV system is described in the application “Interventionin Operation of a Vehicle Having Autonomous Driving Capabilities”, U.S.patent application Ser. No. 15/624,780, filed on Jun. 16, 2017, thecontents of which are incorporated in their entirety.

FIG. 6A illustrates an architecture of a teleoperation system. Ateleoperation system 490 may include the several elements including ateleoperation client 401 (e.g., hardware, software, firmware, or acombination of two or more of them), typically installed on an AV 100 ofan AV system 120. The teleoperation client 401 may interact withcomponents (e.g., sensors 121, communication devices 140, a database412, user interface devices, memory 140, or functional devices, orcombinations of them) of the AV system 120, for example, sending andreceiving information and commands. The teleoperation client 401 cancommunicate over a communication interface 140 (that may be at leastpartly wireless) with a teleoperation server 120.

The teleoperation system 490 has a teleoperation server 410, which maybe located in the AV 100 or in a remote location, for example, at least0.1, 1, 4, 3, 4, 5, 10, 40, 30, 40, 50, 100, 100, 300, 100, 500, 600,700, 900, or 1000 meters away from the AV 100. The teleoperation server410 communicates with the teleoperation client 401 using thecommunication interface 140. In some implementations, the teleoperationserver 410 can communicate simultaneously with multiple teleoperationclients; for example, the teleoperation server 410 communicates withanother teleoperation client 451 of another AV that is part of anotherAV system. The client 401 may communicate with one or more data sources420 (e.g., a central server 422, a remote sensor 424, and a remotedatabase 426 or combinations of them) to collect data (e.g., roadnetworks, maps, weather, and traffics) for implementing autonomousdriving capabilities. The teleoperation server 410 may also communicatewith the remote data sources 420 for teleoperations for the AV system120.

When one or more components of the AV system 42 (FIG. 1) is in anabnormal or unexpected condition (e.g., malfunctions or generates anunusual output), a teleoperation to event may be triggered. Forinstance, a brake malfunctions; a flat tire occurs; the field of view ofa vision sensor is blocked; a frame rate of a vision sensor drops belowa threshold; an AV system's movement does not match a current steeringangle, a throttle level, a brake level, or a combination of them; afault occurs in the software code; a signal strength is reduced; a noiselevel increases; an unknown object is perceived in the environment ofthe AV system; a motion planning process is unable to find a trajectorytowards the goal due to a planning error; a data source (e.g., adatabase, a sensor, and a map data source) becomes inaccessible; orcombinations of them. A teleoperation event may be triggered by anoccurrence or a request. Examples include: a detour, a protest, a fire,an accident, a flood, a fallen tree or rock, a medical emergency, apolice request, a request by an occupant in the AV (e.g., a passengerdoes not like driving behaviors of the AV system), a request by a userof the AV (e.g., a package sender using the AV system to ship packageswants to change a new trajectory or a destination), or initiation by ateleoperator, or combinations of them.

The teleoperation system 490 also has a user interface 412 presented bythe teleoperation server 410 for a human teleoperator 414 to engage inteleoperations for the AV system 100. In some cases, the interface 412may render to the teleoperator 414 what the AV system 100 has perceivedor is perceiving in the real world. Referring to FIG. 6B, exampleteleoperation screens 500 may appear similar to the in-vehicle display341 (which is shown on the user interface device 150 inside the AV 100)and can display a virtual map or a picture display (FIG. 6C) depictingthe vehicle's current position 232, objects 360 in the drivingenvironment, and a portion of the trajectory 198. This rendering can bea schematic map, and can be identical or similar to the in-vehicledisplay 341, with the addition of teleoperation features 502 that allowthe human user 414 to interact with the teleoperation client 401.

The rendering on the teleoperation screen 500 can include enhancementfeatures 220, for example boxes 220 in FIGS. 6B and 6C that highlightidentified objects 360 in the surroundings of the AV 100. Theenhancement features 220 can aid a human user 414 to decide if or how torespond to a teleoperation event or teleoperation request. In someimplementations, the teleoperation server 410 renders an environment ofthe AV system 120 through a user interface to the teleoperator, and theteleoperator can see the environment to select an optimal teleoperation.In some implementations, the user interface that renders the environmentof the AV system 120 to the teleoperator can be a screen, or can bemultiple screens. The multiple user interface screens can be connectedand curved or bent so as to at least partially surround a teleoperator;in some instances the teleoperator can be completely surrounded by arendering of the environment of the AV system 120, creating an immersiveenvironment for a teleoperator. This enables the teleoperator to seeeverything that is occurring in a full 360 degree field around thevehicle, including vehicles and object behind and to the sides of thevehicle that are normally out of the field of view of a person lookingforward. This teleoperator-surrounding rendering can simulate theexperience of a passenger actually riding inside the AV vehicle 100. Therendering can include enhancement features 220, and teleoperationfeatures 502. In some instances, the rendering can be connected to an ARheadset 380 worn by a user inside the AV vehicle 100. The teleoperator'sview can be updated with the passenger's view as the passenger turns hisor head and thereby changes the view of the headset 380.

The teleoperation server 410 may recommend possible teleoperationsthrough an interface to the teleoperator, and the teleoperator 414 canselect one or more of the recommended teleoperations and cause theteleoperations to be sent to the AV system 120. In some examples, theteleoperator uses the interface to draw a recommended trajectory for theAV along which to continue its driving.

Technology Components

FIG. 7 shows an example computer device 600 and example mobile computerdevice 650, which can be used to implement the technologies that we havedescribed. For example, a portion or all of the operations of the AVsystem 150 may be executed by the computer device 600 and/or the mobilecomputer device 650. Computing device 600 is intended to representvarious forms of digital computers, including, e.g., laptops, desktops,workstations, personal digital assistants, servers, blade servers,mainframes, and other appropriate computers. Computing device 650 isintended to represent various forms of mobile devices, including, e.g.,personal digital assistants, cellular telephones, smartphones, and othersimilar computing devices. The components shown here, their connectionsand relationships, and their functions, are meant to be examples only,and are not meant to limit implementations of the techniques describedand/or claimed in this document.

Computing device 600 includes processor 602, memory 604, storage device606, high-speed interface 608 connecting to memory 604 and high-speedexpansion ports 610, and low speed interface 612 connecting to low speedbus 614 and storage device 606. Each of components 602, 604, 606, 608,610, and 612, are interconnected using various busses, and to can bemounted on a common motherboard or in other manners as appropriate.Processor 602 can process instructions for execution within computingdevice 600, including instructions stored in memory 604 or on storagedevice 606, to display graphical data for a GUI on an externalinput/output device, including, e.g., display 616 coupled to high speedinterface 608. In other implementations, multiple processors and/ormultiple buses can be used, as appropriate, along with multiple memoriesand types of memory. Also, multiple computing devices 600 can beconnected, with each device providing portions of the necessaryoperations (e.g., as a server bank, a group of blade servers, or amulti-processor system).

Memory 604 stores data within computing device 600. In oneimplementation, memory 604 is a volatile memory unit or units. Inanother implementation, memory 604 is a non-volatile memory unit orunits. Memory 604 also can be another form of computer-readable medium,including, e.g., a magnetic or optical disk.

Storage device 606 is capable of providing mass storage for computingdevice 600. In one implementation, storage device 606 can be or containa computer-readable medium, including, e.g., a floppy disk device, ahard disk device, an optical disk device, a tape device, a flash memoryor other similar solid state memory device, or an array of devices,including devices in a storage area network or other configurations. Acomputer program product can be tangibly embodied in a data carrier. Thecomputer program product also can contain instructions that, whenexecuted, perform one or more methods, including, e.g., those describedabove. The data carrier is a computer- or machine-readable medium,including, e.g., memory 604, storage device 606, memory on processor602, and the like.

High-speed controller 608 manages bandwidth-intensive operations forcomputing device 600, while low speed controller 612 manages lowerbandwidth-intensive operations. Such allocation of functions is anexample only. In one implementation, high-speed controller 608 iscoupled to memory 604, display 616 (e.g., through a graphics processoror accelerator), and to high-speed expansion ports 610, which can acceptvarious expansion cards (not shown). In the implementation, thelow-speed controller 612 is coupled to storage device 606 and low-speedexpansion port 614. The low-speed expansion port, which can includevarious communication ports (e.g., USB, Bluetooth®, Ethernet, wirelessEthernet), can be coupled to one or more input/output devices,including, e.g., a keyboard, a pointing device, a scanner, or anetworking device including, e.g., a switch or router (e.g., through anetwork adapter).

Computing device 600 can be implemented in a number of different forms,as shown in the figure. For example, it can be implemented as standardserver 620, or multiple times in a group of such servers. It also can beimplemented as part of rack server system 624. In addition or as analternative, it can be implemented in a personal computer (e.g., laptopcomputer 622). In some examples, components from computing device 600can be combined with other components in a mobile device (not shown)(e.g., device 650). Each of such devices can contain one or more ofcomputing device 600, 650, and an entire system can be made up ofmultiple computing devices 600, 650 communicating with each other.

Computing device 650 includes processor 652, memory 664, and aninput/output device including, e.g., display 654, communicationinterface 666, and transceiver 668, among other components. Device 650also can be provided with a storage device, including, e.g., amicrodrive or other device, to provide additional storage. Components650, 652, 664, 654, 666, and 668, may each be interconnected usingvarious buses, and several of the components can be mounted on a commonmotherboard or in other manners as appropriate.

Processor 652 can execute instructions within computing device 650,including instructions stored in memory 664. The processor can beimplemented as a chipset of chips that include separate and multipleanalog and digital processors. The processor can provide, for example,for the coordination of the other components of device 650, including,e.g., control of user interfaces, applications run by device 650, andwireless communication by device 650.

Processor 652 can communicate with a user through control interface 658and display interface 656 coupled to display 654. Display 654 can be,for example, a TFT LCD (Thin-Film-Transistor Liquid Crystal Display) oran OLED (Organic Light Emitting Diode) display, or other appropriatedisplay technology. Display interface 656 can comprise appropriatecircuitry for driving display 654 to present graphical and other data toa user. Control interface 658 can receive commands from a user andconvert them for submission to processor 652. In addition, externalinterface 662 can communicate with processor 642, so as to enable neararea communication of device 650 with other devices. External interface662 can provide, for example, for wired communication in someimplementations, or for wireless communication in other implementations.Multiple interfaces also can be used.

Memory 664 stores data within computing device 650. Memory 664 can beimplemented as one or more of a computer-readable medium or media, avolatile memory unit or units, or a non-volatile memory unit or units.Expansion memory 674 also can be provided and connected to device 850through expansion interface 672, which can include, for example, a SIMM(Single In Line Memory Module) card interface. Such expansion memory 674can provide extra storage space for device 650, and/or may storeapplications or other data for device 650. Specifically, expansionmemory 674 can also include instructions to carry out or supplement theprocesses described above and can include secure data. Thus, forexample, expansion memory 674 can be provided as a security module fordevice 650 and can be programmed with instructions that permit secureuse of device 650. In addition, secure applications can be providedthrough the SIMM cards, along with additional data, including, e.g.,placing identifying data on the SIMM card in a non-hackable manner.

The memory can include, for example, flash memory and/or NVRAM memory,as discussed below. In one implementation, a computer program product istangibly embodied in a data carrier. The computer program productcontains instructions that, when executed, perform one or more methods,including, e.g., those described above. The data carrier is a computer-or machine-readable medium, including, e.g., memory 664, expansionmemory 674, and/or memory on processor 652, which can be received, forexample, over transceiver 668 or external interface 662.

Device 650 can communicate wirelessly through communication interface666, which can include digital signal processing circuitry wherenecessary. Communication interface 666 can provide for communicationsunder various modes or protocols, including, e.g., GSM voice calls, SMS,EMS, or MMS messaging, CDMA, TDMA, PDC, WCDMA, CDMA2000, or GPRS, amongothers. Such communication can occur, for example, throughradio-frequency transceiver 668. In addition, short-range communicationcan occur, including, e.g., using a Bluetooth®, Wi-Fi, or other suchtransceiver (not shown). In addition, GPS (Global Positioning System)receiver module 670 can provide additional navigation- andlocation-related wireless data to device 650, which can be used asappropriate by applications running on device 650.

Device 650 also can communicate audibly using audio codec 660, which canreceive spoken data from a user and convert it to usable digital data.Audio codec 660 can likewise generate audible sound for a user,including, e.g., through a speaker, e.g., in a handset of device 650.Such sound can include sound from voice telephone calls, recorded sound(e.g., voice messages, music files, and the like) and also soundgenerated by applications operating on device 650.

Computing device 650 can be implemented in a number of different forms,as shown in the figure. For example, it can be implemented as cellulartelephone 680. It also can be implemented as part of smartphone 682,personal digital assistant, or other similar mobile device.

Various implementations of the systems and techniques described here canbe realized in digital electronic circuitry, integrated circuitry,specially designed ASICs (application specific integrated circuits),computer hardware, firmware, software, and/or combinations thereof.These various implementations can include one or more computer programsthat are executable and/or interpretable on a programmable system. Thisincludes at least one programmable processor, which can be special orgeneral purpose, coupled to receive data and instructions from, and totransmit data and instructions to, a storage system, at least one inputdevice, and at least one output device.

These computer programs (also known as programs, software, softwareapplications or code) include machine instructions for a programmableprocessor, and can be implemented in a high-level procedural and/orobject-oriented programming language, and/or in assembly/machinelanguage. As used herein, the terms machine-readable medium andcomputer-readable medium refer to a computer program product, apparatusand/or device (e.g., magnetic discs, optical disks, memory, ProgrammableLogic Devices (PLDs)) used to provide machine instructions and/or datato a programmable processor, including a machine-readable medium thatreceives machine instructions.

To provide for interaction with a user, the systems and techniquesdescribed here can be implemented on a computer having a display device(e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor)for presenting data to the user, and a keyboard and a pointing device(e.g., a mouse or a trackball) by which the user can provide input tothe computer. Other kinds of devices can be used to provide forinteraction with a user as well. For example, feedback provided to theuser can be a form of sensory feedback (e.g., visual feedback, auditoryfeedback, or tactile feedback). Input from the user can be received in aform, including acoustic, speech, or tactile input.

The systems and techniques described here can be implemented in acomputing system that includes a backend component (e.g., as a dataserver), or that includes a middleware component (e.g., an applicationserver), or that includes a frontend component (e.g., a client computerhaving a user interface or a Web browser through which a user caninteract with an implementation of the systems and techniques describedhere), or a combination of such backend, middleware, or frontendcomponents. The components of the system can be interconnected by a formor medium of digital data communication (e.g., a communication network).Examples of communication networks include a local area network (LAN), awide area network (WAN), and the Internet.

The computing system can include clients and servers. A client andserver are generally remote from each other and typically interactthrough a communication network. The relationship of client and serverarises by virtue of computer programs running on the respectivecomputers and having a client-server relationship to each other.

In some implementations, the engines described herein can be separated,combined or incorporated into a single or combined engine. The enginesdepicted in the figures are not intended to limit the systems describedhere to the software architectures shown in the figures.

A number of embodiments of the invention have been described. Forexample, although the descriptions in this document have describedimplementations in which the teleoperator is a person, teleoperatorfunctions can be performed partially or fully automatically.

Other implementations are also within the scope of the following claims.

What is claimed is:
 1. An apparatus comprising: a processor; and storagefor instructions executable by the processor to: in connection with atrip of a person in an autonomous vehicle, select a specific location atwhich the person will be picked up for the trip or a specific locationat a destination of the trip; and present through a user interface of adevice visible information that depicts the specific location.
 2. Theapparatus of claim 1, wherein the specific location comprises a specificphysical location on or near a road.
 3. The apparatus of claim 1,wherein the specific location comprises a specific physical locationidentified prior to a time when the person requested the trip.
 4. Theapparatus of claim 1, wherein the visible information comprises a realimage of the specific physical location.
 5. The apparatus of claim 1,wherein the visible information comprises a real image of an autonomousvehicle presented with a real image of the specific physical location.6. The apparatus of claim 1, wherein the specific location is notidentified by a street address.
 7. A mobile device comprising: adisplay; a processor; and an app or a browser to cause the processor topresent on the display a depiction of an actual specific location atwhich a person will be picked up by an autonomous vehicle for a trip ora specific location at a destination of the trip, the actual specificlocation having been determined prior to a request by the person for thetrip.
 8. The mobile device of claim 7, wherein the specific locationcomprises a specific physical location on or near a road.
 9. The mobiledevice of claim 7, wherein the specific location comprises a specificphysical location identified prior to a time when the person requestedthe trip.
 10. The mobile device of claim 7, wherein the visibleinformation comprises a real image of the specific physical location.11. The mobile device of claim 7, wherein the visible informationcomprises a real image of an autonomous vehicle presented with a realimage of the specific physical location.
 12. The mobile device of claim7, wherein the specific location is not identified by a street address.13. An apparatus comprising: a processor; and storage for instructionsexecutable by the processor to: receive through a user interface of adevice a signal from a user with respect to a trip of the user in anautonomous vehicle, the signal indicating a feature of the autonomousvehicle that can be controlled in response to the signal from the user;and determine an action to take with respect to the autonomous vehicleto respond to the signal from the user by controlling the feature of theautonomous vehicle.
 14. The apparatus of claim 13, wherein the featureof the autonomous vehicle that can be controlled in response to thesignal from the user comprises a temperature inside the autonomousvehicle.
 15. The apparatus of claim 13, wherein the signal received fromthe user comprises a temperature inside the autonomous vehicle.
 16. Theapparatus of claim 13, wherein the feature of the autonomous vehiclethat can be controlled in response to the signal from the user comprisesa passenger capacity of the vehicle.
 17. The apparatus of claim 13,wherein the signal received from the user comprises a number ofpassengers for the trip.
 18. The apparatus of claim 13, wherein thefeature of the autonomous vehicle that can be controlled in response tothe signal from the user comprises a state of an entertainment systeminside the vehicle.
 19. The apparatus of claim 13, wherein the state ofthe entertainment system comprises at least one of a type ofentertainment content.
 20. The apparatus of claim 13, wherein the stateof the entertainment system comprises an identifier of a broadcaststation.
 21. The apparatus of claim 13, wherein the signal received fromthe user comprises an identification of a source or type of content forthe entertainment system.
 22. The apparatus of claim 13, wherein thefeature of the autonomous vehicle that can be controlled in response tothe signal from the user comprises accommodation of a child who requiresa child seat.
 23. The apparatus of claim 13, wherein the signal receivedfrom the user comprises an indication of a child to be present on thetrip.
 24. The apparatus of claim 13, wherein the feature of theautonomous vehicle that can be controlled in response to the signal fromthe user comprises accommodation of packages for the trip.
 25. Theapparatus of claim 13, wherein the signal received from the usercomprises of information about packages for the trip.
 26. The apparatusof claim 13, wherein the instructions executable by the processor are toselect an autonomous vehicle from among two or more available autonomousvehicles based on the signal from the user.
 27. The apparatus of claim13, wherein the selection of the autonomous vehicle is based on at leastone of the passenger capacity, the package capacity, or the availabilityof a child car seat.
 28. The apparatus of claim 13, wherein theprocessor and storage are part of a central AV system.
 29. The apparatusof claim 13, wherein the instructions executable by the processor are tocommunicate the action to the autonomous vehicle.
 30. A mobile devicecomprising: a display; a processor; and an application or a browser tocause the processor to present on the display at least one option tocontrol a feature of an autonomous vehicle for a trip, the featurecomprising at least one of a passenger capacity, a package capacity,availability of a child car seat, a temperature inside the vehicle, or afeature of an entertainment system.